Measurement Uncertainties of Three Score Distributions and Two Thresholds with Data Dependency

The National Institute of Standards and Technology conducts an ongoing series of Speaker Recognition Evaluations (SRE). Recently a new paradigm was adopted to evaluate the performance of speaker recognition systems in which three distributions of target, known non-target, and unknown non-target scores, as well as two thresholds were employed. The new detection cost function was defined to be an average of the two weighted sums of the probabilities of type I and type II errors corresponding to the two decision thresholds. In addition, data dependency due to multiple use of the same subjects is also involved. The data were reorganized into a two-layer structure in view of the data dependency and the probability theory. Then, the uncertainties of the detection cost functions were computed using the nonparametric three-sample two-layer bootstrap method. Comparing these results with those calculated by using all the raw data and the nonparametric three-sample bootstrap method with the i.i.d. assumption, the measurement accuracies, i.e., the detection cost functions, have changed little; but the measurement uncertainties, i.e., the standard errors of the detection cost function, have improved as a result of taking account of the data dependency. Forty speaker recognition systems were used as examples. Index Terms – Metrology, measure, accuracy, uncertainty, bootstrap, data dependency, speaker recognition. This publication is available free of charge from http://dx.doi.org/10.6028/NIST.IR.8025